All right, I'm Zaz, and it's a great pleasure to be back here at DEF CON China 1.0.
My talk is going to be on hacking driverless vehicles, and I'm also going to be including
connected vehicles in that, because basically unmanned vehicles are just another Internet
of Things device, except they're an Internet of Things device that's covered in sensors
and that's able to move around in the real world and cause all kinds of mayhem.
So, I'll be looking at all of that stuff, and one important distinction that I want
to make right from the beginning is that I'm not going to be looking at attacks that include
a network foothold on the platform.
If you have a network foothold on the platform, you can do anything you want.
That's well known.
I'm going to be looking at ways that you can attack these vehicles from outside the
system.
A little bit of background about myself.
I have an autonomous robots background.
I have a PhD.
I have a PhD in robotics, and I am best known perhaps for a TV show that I did with another
speaker that spoke at 10 a.m. this morning, Joe Grand.
The show was called Prototype This, and in the course of that show, we actually prototyped
a number of autonomous systems.
For example, we did an unmanned aerial vehicle that delivered life preservers out to a swimmer
in the ocean, and we learned a lot about the various failures of aerial platforms and
how to iterate on those platforms as part of that.
We also turned our hands to ground vehicles, looking at one of the principal problems in
the autonomous ground vehicle space, the efficiency of pizza delivery.
So we did two vehicles there, a local vehicle for delivering pizzas on the streets, and
a long-range vehicle for long-range pizza delivery, which included the first-ever autonomous
crossing of a highway bridge in the United States.
So this was pretty serious work.
But I...
You know...
Although...
Autonomous vehicles seem like a new and shiny thing, I want to take you back to the
sort of origins of autonomous vehicles, which weren't, in fact, in the United States.
The first stuff was done in Germany in the 1980s, 1986, a vehicle called VaMors that
was done by a pioneer of autonomous vehicles, Ernst Dickmanns.
And in 1995, they drove an autonomous vehicle from Munich to Copenhagen at up to 175 kilometers
per hour.
On the Autobahn, using nothing but vision, no GPS, no LIDAR, just using cameras.
So this stuff is pretty old, right?
So this is kind of like...
What's this talk about?
It's about hacking Windows 3.1?
No.
A lot of...
There's a lot of new stuff being done in the space.
So I'm just going to do a quick summary of what's happening in the autonomous vehicle
area in Asia, because America is getting all the press, but a lot of interesting stuff
is happening here in Asia.
So Singapore.
They're looking to be a regional leader.
They have autonomous bus services that they're planning to schedule by 2022.
But China, right here, is the place that's positioning itself to be a technology leader
in unmanned systems.
So they're doing autonomous bus testing in Shanghai right now.
In Guangzhou, public road testing since last year.
Here in Beijing, Baidu is testing on 105 kilometers of suburban roads, and they have a partnership
with Volvo.
So they're looking to deploy level four autonomous taxis.
Fully driverless taxis by 2021.
Pretty exciting.
So there's a revolution that's going to happen in driverless systems because of so many advantages.
The energy efficiency of not having to haul a human driver everywhere you go, the time
efficiency of not having to deal with the needs of that driver, and the new applications
that it opens up.
So lots of areas in the civil space, obviously transportation, also oceanography.
The ocean is a dangerous and boring place for the most part.
It's a great place to send robots.
Mapping, of course.
Filmmaking already.
Autonomous drones have taken off there.
Engineering tasks like power line and structural inspections.
And logistics.
Delivery.
Right?
That's something that has to happen everywhere.
So in particular, in terms of logistics, don't just think about the air and the ground.
The sea is a very important part of the logistics chain.
And unmanned cargo shipping is the future.
Because the accidents are mostly good.
And that's what's going to be caused by the humans.
75% of maritime accidents caused by human error.
And a major technical challenge that has to be overcome, though, before we can deploy
our unmanned systems is dealing with these long voyages, months at a time, and having
hardware failure.
A lot of the reason you have a human crew is to fix things that break.
But here's an example that's just been announced.
The Kongsberg company has announced they're going to deploy this boat called the Yara
Birkeland.
Which is going to be an electric, zero-emissions vehicle.
And it's going to be an electric, zero-emissions vehicle.
With autonomous capability.
And so this will just give you an idea about how these systems are likely to be rolled
out.
It's going to replace 40,000 truck trips a year.
So they're going to start in 2020 as a manned vehicle.
They're going to drive this boat back and forth, delivering its things.
2021, it's going to start down-crewing.
And letting the robotic systems take over.
Preparing for fully autonomous operation in 2022.
So that's the kind of thing that you'll see out there in the real world.
.
But the priorities of the industry in the civil space are agriculture and self-driving
cars.
And the things that are in the way why we don't have our roads filled with driverless
cars right now when Dickens was doing this stuff in the 80s and 90s, is the fact that
we have shared infrastructure.
We have to share the roads and the airspace and the oceans with human piloted vehicles.
And the public acceptance.
The perception of safety and robustness.
So. . . . . . . . . . . . . .
and robustness, let's talk about how these systems fail.
So I'm going to look at two case studies of classic failures.
Number one, there was an event held in 2004 called the DARPA
Grand Challenge.
That was an autonomous robot road race from Los Angeles
to Las Vegas, the home of the original DEF CON.
And this was a big deal.
The prize was $1 million.
And a lot of teams put a lot of work and a lot of money
into trying to be the first to win this road race.
And the favorite for this race was a vehicle called Sandstorm
from Carnegie Mellon University.
They developed all kinds of new sensors.
They loaded it up with everything that it had to offer.
The vehicle took off, and everyone was super excited.
This was the one that everyone thought was going to win.
Well, what happened?
This vehicle got a few miles along the course,
and then it veered off the road, crashed, rolled over,
and caught fire.
So.
There it is, on fire off the side of the road.
What went wrong?
Everyone was very disappointed.
Well, what went wrong was the fact
that they knew the course a couple of days in advance,
and they manually walked the whole course.
They had a perfect GPS map.
But when they actually went to run the race,
they had to set the weighting to trust
the sensors versus the GPS.
And they trusted the sensors too much.
The sensors got a wrong reading, drove the car off the road.
So it's a constant battle to know,
what the robot actually knows.
And the correct estimation of that state
is the key to decision making.
And so a successful exploit of an autonomous vehicle
will most likely subvert the state estimation process.
Here's another more recent big failure.
This was a Tesla in autopilot mode.
This is from last year, March 2018.
A huge accident that unfortunately
was fatal for the driver.
What went wrong?
So the Tesla was in autopilot mode.
And so it had two types of autonomous behaviors
that it was doing.
Number one, dynamic cruise control.
It was following the vehicle in front.
Number two, automatic steering with lane following.
It was following the lanes and staying in its lane.
It got to a fork in the highway.
And it could have just kept following the vehicle in front,
and it would have been fine.
Instead, it followed bad lane markings, and it crashed.
It crashed right into the barrier between the two roads.
At 120 kilometers per hour, the crash attenuator
that was there on the highway was
supposed to stop that from being a terrible crash.
But it was previously damaged.
It wasn't fully functional.
And so it was a bad accident.
And as I said, the driver unfortunately died.
And this all came down to one decision.
The vehicle selected those poor lane markings instead
of following the lead vehicle.
So autonomous vehicles is an area that's
full of fragile decision making and edge cases that
have to be decided in a split second.
So let's take a look at the logic structures
inside an autonomous vehicle.
There's a hierarchy of behaviors
that operate at different speeds and at different priority
levels.
So at the bottom, you have your control loops,
the maintenance of stability and operation of the vehicle.
Above that, collision avoidance.
So making sure once you have control of the vehicle
that you can control it not to run into things.
Then you've got your navigation and localization,
getting where you want to go.
And at the top level, your task planners and your reasoners.
If you're a taxi, getting to the right place
and then billing the customer, delivering pizza, so on,
which I'll get to in a sec.
So if you're attacking one of these systems,
then you might want to attack lower level in the stack,
because they defeat everything above that.
However, because those lower level things are more mission
critical, then more engineering effort is spent on guaranteed
robustness.
So they may be a harder target.
You've got your juicier but harder targets,
just like attacking many types of infrastructure.
So two quick examples from the very beginning of this talk.
The life-saving drone first.
In its control loop level, it has its PID loops
on its autopilot tuned to be able to cope with wind gusts
and warm rising air, cold descending air, and so on.
Then the collision avoidance level, absolutely nothing.
This is a big problem for aerial vehicles,
is that there's very little sense and avoid in place.
Navigation and localization.
That was simple waypoint circuit,
going from waypoint to waypoint.
And then the mission reasoner was
to set up this dynamic bombing run
to drop the life preserver to the person in trouble.
So this is a system that's extremely vulnerable
to collision, because there's no collision avoidance.
And its high level logic depends on one single sensor, GPS.
So if there's any problems with that sensor,
the system's going to have problems.
Pizza delivery.
I'll look at the local pizza delivery,
because it's kind of more interesting.
The control loop level, this is a two-wheeled vehicle.
So it has to do balancing and dynamic weight shifting.
It's got to deal with the fact that the weight on top
changes when a pizza is removed.
At the collision avoidance level,
that's what this system's all about.
Dynamic obstacle discrimination and avoidance,
using the LIDAR as a primary sensor.
Navigation and localization.
Route planning using a map that the system builds itself,
with a process called SLAM.
Simultaneous localization and mapping.
And then the final high level reasoner
is to dispense the pizza to the correct person,
using the credit card as an authentication mechanism.
So this is a system, because it's
doing its dynamic obstacle avoidance
and it's building its own map, it's
vulnerable to all types of redirection, trapping,
and map confusion attacks.
So let's take a look at the suite of sensors
that you might encounter in an autonomous system.
There's a basic difference in sensor types.
You have your active sensors that send information out.
And reader return.
And the passive sensors that just read in information
from the environment.
So commonly, you'll find GPS.
And they're usually mounted on the roof,
so they have an uninterrupted view of the sky.
LIDAR, also often mounted on the roof.
That's a laser range finder, which we'll get to later.
You've got cameras for doing vision.
Often a millimeter wave radar, usually
on the front and back of the vehicle.
Ultrasonic transducers are common for parking.
Usually have a digital compass.
It's always nice to have an absolute
direction reference.
An inertial measurement unit, so that you
can measure acceleration and rotation.
And then wheel encoders, so that you can get
some kind of absolute odometry.
And then for other types of vehicles, for example,
under the water, you might have sort of more esoteric sensors
that are designed to operate in that domain.
So a Doppler velocity logger, for example, under the water.
Scanning sonar.
And pressure transducers, often used both under the water
and in the air to give you, for example, a barometric altimeter.
So sensors have a number of sources of uncertainty, right?
They constantly have noise that they have to deal with.
They drift.
So that's where they, over time, get
a reading that's further and further away from the true reading.
And then you have latency problems.
So you have some time delay between when
you read the sensor and what the actual value was.
So the real world may have changed
during that latency period.
So you have to model that noise.
You have to model that uncertainty
under a series of assumptions.
When you have more than one sensor, that's great.
Because then you can do sensor fusion.
So fused or registered data can be
more useful than separate individual data
from one sensor alone.
However, what do you do when those sensors disagree?
This is very topical right now because of an automation
failure that's been in the news, the Boeing 737 MAX,
which had two angle of attack sensors,
but they were only using one of them.
They chose not to do sensor fusion for whatever reason.
Boeing has never explained publicly
why they chose not to do sensor fusion.
But it may have been, what do you
do when the sensors disagree?
And what they're going to do now with the new 737 MAX fix
is to display a cockpit light that tells you, tells the pilot,
when the sensors are disagreeing.
So again, your robustness of your system
may come down to how good is your system at discounting
one malfunctioning or one intentionally spoofed
or hacked sensor.
That was Boeing's problem.
So let's take a look at our general classes
of sensor attacks.
Two basic kinds of attacks.
We've got denial, just like denial of service.
That's preventing the sensor from getting any good data.
Or spoofing.
That's when you're deliberately causing the sensor
to retrieve specifically bad or incorrect data.
So you've got a basic attack mode choice as well
from the attacker's perspective.
Do you attack sensors instantaneously?
Do you give them immediately bad data?
Or do you do a more subtle attack,
where you give it aggregated bad data
and you build up a poor model over time
to cause it to make bad long-term inference?
So I'm going to take a look at all these various sensors
and how we do both kinds of attacks on them.
So GPS.
Denial is very easy with GPS.
We can jam it.
You can buy online a GPS jammer.
Check your local GPS.
You can also build your own area for legality.
But China is a place that manufactures
many of these GPS jammers.
You can also build your own.
Schematics are available online.
And they just throw a bucket of radio noise out.
And they just prevent the receiver
from receiving valid satellite signals.
Because the actual signals that come from the satellite
are very weak.
Spoofing is also possible.
So basically, you're sending out valid GPS signals
with a radio at a higher power than the genuine signals
from the satellites.
So here's an example of that from the UT Austin Radio
Navigation Laboratory, which is one of the first places
to really popularize the possibility
of active spoofing of GPS.
So they're demonstrating here that to take over
the GPS of a vehicle, you send out a valid signal first.
You take over the tracking points of that system.
And then.
You move your fake signal away.
And by doing that, you can affect the controller
on board the vehicle.
So for example, an aerial platform,
if you send your fake GPS signal,
send out a signal that makes it look
like the vehicle is going up, then the controller on board
will drive it down to compensate.
So here's an example on a real aerial vehicle.
They're about to send a fake signal
and tell the vehicle it's going up.
And so the controller is going to tell the vehicle it's going up.
It's going to drive it down.
And if the safety pilot didn't take over,
this vehicle would drive itself right into the ground.
This is something that the Iranians said
that they did when they captured
this American military drone.
They captured this RQ-170 drone,
and they claimed that they used GPS spoofing to bring it down.
That's almost certainly not the case,
because everybody knows that GPS is very spoofable.
The military knows that.
They have their own encrypted GPS signal.
But they don't use GPS as a primary sensor
because they know that it's jammable and spoofable.
What happens, though, in a real system
when a GPS system causes a problem?
So here's an example from the second DARPA Grand Challenge.
This is a vehicle called Sandstorm.
And we can see a straight road,
but it's pulling it off the road.
That's what'll normally happen when you have a vehicle
that's got multiple sensors and it's getting bad GPS data,
is that it won't immediately...
Go off where you want it to go. It'll sort of be trying to pull it back onto the track
Here's another example from the doppelganger challenge of the perils of GPS only navigation
This vehicle completely fails to do collision avoidance because it's getting bad GPS data and that's pulling it off the road
It's not just cars that use GPS
in particular
For a ship's GPS is often the primary navigation sensor because out in the middle of the ocean you have no other navigation references
Especially during the day when you can't use the stars for for navigation reference
So that same lab the UT Austin radio navigation lab decided well, let's see if we can take over a super yacht
So they went they found a rich person and they said hey
can we borrow your yacht to see if we could potentially steer this thing off course and
sure enough
they
, uh
had their attack system on board the yacht and
Instead of sending a drift signal up or down like you would send an aerial vehicle. They they changed the angle
of the
Direction of GPS travel by just three degrees
Because three degrees over the thousands of kilometers that a boat might travel across an ocean can put that boat in a very very different
Location so they were able to use their spoof GPS to send a different navigation track and cause that ship to go off course
There's the radar
Image from that ship to prove that that was possible
so
Originally this might have been an expensive attack
But now SDRs have made this attack very very inexpensive the first one to be presented at a hacking conference
Was this one using the blade RF so for about a thousand dollars?
This was the Quihu 360 unicorn team at DEFCON 23. They showed that
for about a thousand dollars using a blade RF and
They could
spoof GPS and that has become even easier since then as SDRs have come down in price
Also, they went to car showrooms and they showed that the in-car navigate navigation system could believe that the car was in the middle
of a lake somewhere
Now we can do this with a hacker f1 for about four hundred dollars
You do have to modify it. So you need to get that the built-in oscillator in a hacker f1 is not
low drift enough to do this attack
Reliably, you need a 10 mega megahertz TST CXO low drift oscillator
And you make a little daughter board for that
So here's the OSH Park images for that daughter board and that shows it installed on the hacker F
Then you go to NASA's site
They provide the GPS ephemeris data all of the details of where the satellites are for the previous day
You can then so you can spoof for any any day in the past by doing that and then you can also
Take the time to spoof in the future and the then you use the open-source project GPS SDR sim to
Convert that ephemeris data into active
Interactive GPS signals that are then transmitted out through the hacker F. So
I'm actually doing that in this auditorium right now. Unfortunately, it just stopped. So I'm just started again. It may take a
little while to
get the signal
but we can
I've got the maps here. We can see here my GPS device if we can switch to the camera
Can we grab the camera view over here
Hopefully you can see that I'm getting full satellite signals on this device
It hasn't quite picked it up yet because I just restarted the system
But it's getting them you can see there. It's got like quite a quite a few satellite views already and
then
This iPhone has already got the signal and I don't know if you can see it there
If there's any way to zoom in on that, but right now and you can check this on your own phones
I recommend turning Wi-Fi off before doing this so it's only relying on GPS
But hopefully we can see here that I'm broadcasting GPS right now
That's putting everyone in this room in the middle of the White House in the United States
So I can see the signal
Here's here's my rig here with a hacker f1 connected to an hour Raspberry Pi connected to my computer
So you can take down UAVs with this technique very easily if they're a if they're a little commercial drone that has no Flyzone capability
Here's a video that I did of make forcing a
Mavic-DGT
Mavic drone to land
so you can see here, I'm simulating the GPS of the middle of an airport and
that's going to
Force it to land and
It'll take a few seconds, but you'll see that the no-fly zone
Capability has been activated. Yeah. Well, I don't know why it keeps transitioning to the next one
Anyway, the drone lands because it thinks it's in an airport, which is a no-fly zone
moving on to
LiDAR, which is a laser rangefinder. These were originally industrial monitoring sensors
They were used for factories and so on to keep people out of places. They weren't supposed to be there are mechanically scanned laser
And they're primarily used in robotics for collision avoidance and map building
So in order to deny LiDAR access
You can actively overpower them with a strong laser
Which or you can prevent the return signal because it needs that signal to bounce off something and come back?
You can make it so that signal doesn't come back
for spoofing them
You can manipulate the absorbance of the reflectivity of items
In the environment so they look like something else
Or you can actively spoof them and I'll discuss all of those methods here
You can see the kind of point cloud map that the LiDAR builds
So as a 2d sensor, right?
There's this laser that's being scanned back and backwards and forwards or left, you know one way to the other if they're highly
Emission-dependent, so if you have a platform that tilts like this two-wheeled vehicle
The kind of thing that the LiDAR sees it depends a lot on the angle that the vehicle is at
So that means inclines can look like obstacles and it means it may miss low obstacles and discontinuities
It's an active emission sensor it's sending out information so it can only see something that bounces a signal back
So if it doesn't see something it thinks nothing is there it has no information about it?
So think about it everything over the horizon returns nothing. It's just empty space
So most of the world returns no data to the LiDAR you actually even though you get a dense point cloud of things that are there
Looking 360 degrees in every direction. Most of its got nothing
so
Also absorbent things look like nothing and transparent things the laser goes through them
So that means we can do some physical attacks so we can paint absorbent material on a wall
Just like the old Wile E. Coyote gag and the vehicle might try and run right into it
We can also use obstacles that let light through and the LiDAR won't see them
Reflective things can cause problems for the later for the LiDAR
For example puddles of water on the ground
This is something that causes all kinds of difficulties for real-world vehicles
So that means that things that are far away the laser can bounce off
reflect back and they look like
They're close
So a tree in the distance can look like a tree coming out of this puddle and it means if you don't get a return
It's indistinguishable from a hole in the ground, right?
You see a big gap in the in the road. Is that just a puddle or is it a huge hole?
You can't really know
These techniques by the way
Are known by insurgent groups that are trying to deal with military applications of this technology
So here's a a document that was captured from an al-qaeda in the rate in the Arabian Peninsula
group
And it mentions GPS jamming with a Russian made recall system and it mentions using reflective materials to thwart laser designators
So these are valid adversarial techniques
Reflectance though in the case of unmanned vehicles is also a feature because one thing that reflects is
Road line markings. It's actually a very robust method to take those holes in your LiDAR point cloud and do shape matching on them and
Determine and use them to find road markings
so
That means that's a that's a way that we can do an attack that fakes
Road-markings in a way that the robot sees them but a human like safety pilot doesn't see them, right?
So we can paint black-on-black reflective material that a human can't see but we can cause
Unpredictable behavior from the vehicle
or we could just send some jokes in
You know for the people back at the mapping zone. I'm sorry. I went too fast on that slide
But you know to send things back to home base
Solid looking objects are indistinguishable from actual solid objects because the laser just knows when it bounces off something so we can have lightweight obstacles
and
In the case of denial we can use a strong laser source to overpower the LiDAR in a certain area
So here's some research from keist in Korea showing that if we point a strong laser
At a multi beam velodyne so we can see here all those horizontal stripes are the individual beams of this multi beam LiDAR and
By pointing a strong source at it
We can blank out part of the area of its scan so we could hide an obstacle by overpowering the LiDAR
Selectively in a spot, but it gets more interesting than this
We can actually spoof returns by using a property of the way that most of the LiDAR is a built so here's an example
of two very popular lighters the the velodyne pucks and
Both of those things have curved glass on them because that way they can scan 360 degrees
Because we don't want to have you know
Sharp edges to the glass which are going to be difficult to transmit the laser through
But we can use that refraction to move on our false points by modulating the the power of
the Nile's
our spoofing laser source, so
That means if we were attacking a vehicle from the side
We could make it think there was something in front of it
So we don't have to put our fake returns in line between us and the target we can make them show up somewhere else
so
This is using exploiting the fact that the curved glass refracts the light inwards to a different location
And then we can modulate that with source strength
So here's an example of using a weak source and we can see the false returns are
between the
Spoofer and the actual light are but then increasing the strength of our source from the same location
Places our fake returns off to the side
Now we can also do active spoofing we can do a kind of a relay attack
But this is a much more complicated attack this involves characterizing the behavior of the remote laser
so we have to watch for a while see what types of pulses it puts out what the frequency of pulse is and
Then we can start sending predictive returns from the spoofing source
this also gives us great
control over where our fake returns are located, but this means the timing is critical so you have to very very accurately characterize
The pulse rate of the source and its location, but we can see here
The spoofer is located to the right of the sorry to the left of the
Target light are but we're inducing fake dots
to the left or changing the timing now we can put the dots between us in it so
that that gives us control not of the full area of the full XY plane but it
does give us control of the distance from the from the target of the fake
returns now a system that's out there in a large number of vehicles is the Tesla
autopilot because they've put that into a lot of their vehicles and they're able
to update that as an over-the-air update and Tesla's philosophy here is to
only use cameras they do use a little bit of millimeter wave radar for long
distance collision avoidance but mostly their theory is we do everything with
our eyes when we drive so our cars should be able to do everything with
cameras we already saw one example of where that's caused a fatal accident but
so let's talk a little bit about cameras so cameras are used for specialized
object detection
including
signs traffic control features and lane markings sometimes we'll use stereo
cameras to get a depth map so to get a noisy version of what the lidar gives us
and sometimes we'll use it to fuse with a lidar in a method called colorizing
the lidar that's getting color information from the camera and and
melding that with the distance information from the lidar this is
useful for this reason
this is a video from the second DARPA Grand Challenge this is the
vehicle that won that one the second one actually did have a winner it was a
vehicle called Stanley from Stanford and they won because they were able to
drive extremely fast and the way they did that was to use the lidar to give a
sense of drivable versus non drivable and they matched color to drive ability
so that was a way they were able to look further out than the lidar could look by
saying okay this is what color the the drivable road is the rest is non
drivable and being able to drive at speed
cameras are very easily denial of service you can dazzle them with a bright
light and then spoofing is all the ways that we know about from the ways that
our own vision is fooled camouflage techniques assumptions about what color
things are and then in the case of stereo repeating patterns really
confused the stereo system i want to say very briefly about something about all
the research that's come out lately about spoofing deep learning camera
recognition models
this stuff right now is kind of overblown but I just mentioned it for
completeness so there's a very famous paper on adversarial examples for stop
signs so they trained up a recognition model on stop signs and then they were
able to add some very small amounts of black and white tape that prevented
their model from recognizing the stop signs of course feature-based sign
detection is pretty robust and you probably wouldn't train a deep learning
model to find stop signs in the first place
there have been other examples whereby adding adversarial noise to the incoming
camera image you're able to get it to make false recognition so in this case
the pedestrians are missed when you've added some very fine invisible to the
human adversarial noise of course if you have the ability to add adversarial
noise to the camera signal post acquisition that means you have a
foothold on the platform there's lots of other things you could do generally
these techniques are white box techniques so they're these are
researchers that have trained their own model they
know how the model works and so then they attack it based on things that they
developed to fool their own model and they don't tend to work reliably in the
face of parametric distortion so when you have different angles different
amounts of glare and things like that different lighting conditions they don't
work so well here's an example of something that was pretty robust it was
robust enough that they were able to 3d print objects and then they could show
it to the camera in any orientation and it would still work these are the
adversarial turtles from MIT so this
for example is a turtle that's obviously a turtle but it has patterns
printed on it so that the cat the vision system thinks it's a rifle but again
these techniques are not robust and they often involve the injection of noise
that requires a foothold on the platform I mentioned that for
completeness as I said cameras in general involve fragile fragile
discriminators so here's an example from just this year from the Tencent Keen lab
they were looking at Tesla autopilot they were looking at Tesla autopilot they
found some results that are not entirely surprising but it's proof of
what I'm saying so for example they added white tape to the road here to
blur a lane marking and they showed that the system successfully didn't pick
up a lane marking so this is real-world blurring to prevent the autopilot from
seeing a lane and then the converse is also true so they were able to add just
a few dots to the road you can see them there in the upper right and just those
three fake markers were able to cause the vehicle to make an unplanned lane
change right so just like that fatal accident this is something that requires
a small amount of real-world preparation but can cause a very bad accident
moving on to the millimeter wave radar this is the system that you find at the
airport often to scan you and see under your clothes primarily used for
collision avoidance they're a lower resolution than the lidar
because
they're a very low resolution and most things are very very reflective to the
radar so it returns a lot of noisy data so you can do not deny them with a radar
jammer or with chaff which are little pieces of metal that is spat out that's
what the military uses to fend off radar guided missiles very flat reflective in
the RF domain things like overhead signs cause problems for the radar so this is an example of a Bosch millimeter wave radar sensor that's used to detect the radar.
used in the Tesla, and this was shown to be jammable at DEFCON 24. So if you had an oscilloscope
and a signal analyzer, a signal generator, and a harmonic mixer and frequency multiplier,
it's about $100,000 worth of equipment, then you could jam the Tesla autopilot and cause
it not to see a vehicle that was in front of it. So a very expensive attack, but it's
possible. They theorized spoofing and relay attacks, but did not perform them.
Moving on to the inertial measurement unit and the compass. Here's an example of three
such systems at various price points. There's a wide, wide range of IMUs available from
very cheap hobbyist level ones for a few dollars all the way up to many, many thousands of
dollars. These are very, very robust, or they can be very, very robust. So they are used
at the price point.
They have a primary navigation sensor for some systems, such as certain underwater
vehicles or such as some military systems. Because of the high fidelity, if you're going
to shell out for, say, an IMU from a Boeing 777, you can expect 0.1% of the distance
traveled as cumulative error. So that means if you go 300 kilometers, let's say, under
the ice, when you pop up to the surface, you're 300 meters away from where you think you would
be, or within a 300-meter radius. So pretty accurate.
Not much you can do in the way of attacks. Very, very difficult to interfere with. You
can do physical attacks with magnetic fields and exploiting thermal drift if you have physical
access to the platform. One particular interesting physical attack on a particular type of readily
available IMU is the acoustic attack. This is pretty exciting. This is because those
cheap MEMS IMUs actually have physical components in them. So if you have an IMU, you can actually
see those particles. And then the last thing that we'll be talking about is also called
a microelectromechanical. There's a vibration element in the MEMS gyroscope. And there's
a vibrating element in the MEMS gyroscope, and it has a resonant frequency. So this
is what a MEMS microelectromechanical system gyro looks like. And you can disturb that
with an external acoustic source. So this is similar to attacks that have been publicized
recently on hard disks, the, what they call the screaming in the data center attack. So
you blast some noise at a hard disk and it vibrates the read head. And you can cause
the hard disk to not be able to read or even cause physical failure of the drive.
Similarly you can do this to a MEMS gyro. So here's an example from KAIST in 2015 where they
took a real multi-rotor platform and they blasted acoustic noise at it and you can see here this is
the control outputs showing that when the noise was blasted the control went nuts because it was
trying to deal with the fact that these MEMS gyros were vibrating at their resonant frequency from
the noise. And so they were able to successfully crash at whatever distance you can blast that
level of sound pressure, a multi-rotor craft. Next sensor wheel odometry. These are physical
encoders that measure the rotation of the wheel. So here are some that you might see on a commercial
driverless vehicle at present. These are good to know what your real speed is and when the
vehicle has actually stopped. And here's an example from our show
prototype this when we were doing our bridge crossing on one of our attempts the vehicle
took this turn too closely and it actually scraped into the barrier and it knocked its wheel encoder
off. You can see it there knocked off. At that point the car got very confused. It wasn't able
to get ground truth odometry and it wasn't able to proceed. So a physical attack that knocks that
off can cause stopping of the vehicle or other unpredictable behavior. So ways you could attack
the wheel odometry is to change the diameter of the wheel to make the surface slippery
and to scrape them off to force the car to go close to something and physically remove them.
Another sensor that I'm including really just for completeness here is the ultrasonic sensors.
These are most commonly used for automated parking. So this is stuff that happens at a
very slow speed. You can attack them in all the same ways as the other remote sensors. You can jam
them. So here from Defcon 24 sending out ultrasonic noise to prevent them from getting a reading.
You can send fake pulses back to make it look like something is close when it's not really there.
And you can also do an interesting thing because it's in the acoustic
domain you can do cancellation. So if you characterize the sensor you can send an
inverse pulse and cancel out the real returns that are coming from the sensor. Again it's a
low impact attack quite literally because this is only happening at parking speed. So you know if you
have a vehicle in a very low speed you might have a lot of free space to hit the road and and
Now let's talk about components. The first one is going to be James Bond and producer vehicle
that was going to be able to have various attacks involved against autonomous vehicles.
You would have a gps jammer of course.
You have smoke and dust and vapor to interfere with the LIDAR beams
lightweight decoy obstacles
Radar chaff
transparent things to drop down and puncture the tires
And of course no bond vehicle is complete without an oil slick and it works a little differently in this case
You're fooling the wheel odometry
And then based on more recent research
You have your active LIDAR jammer and spoofer active radar jammer your acoustic blaster
And of course your fake lane markings and your adversarial total dispenser
So I'm just gonna quick do a quick time check here and see where I am on time. All right
So I'm gonna move pretty quickly through these next things just to try and get through to the last stuff
But the map is very very important
There's a great emphasis these days on pre-acquired map data
And that's primarily because some of the big developers of autonomous system autonomous vehicles
like Google they already had a lot of mapping data and
So here you can see a LIDAR acquired point cloud
You can see how detailed your map can be so detailed in fact that it's often considered by designers to be the truth
And that's because it's very very useful if you know where everything is in the world
You don't have to do as much work on recognition things that are hard to spot like traffic lights like vegetation
You can just know where they are speed bumps, for example
You don't need to see them
If you know exactly where they are and if you know exactly when you are
Here's a quick example on traffic lights
You can have a camera that's programmed to just look where you know the traffic lights are
And then you can check the status of the lights easily. You don't need to detect lights
You just need to look in the right place and detect state
But that means fake traffic lights
Won't be detected by your system, but they will be obeyed by human drivers. These are shared roads
so they create a difference between
stuff that the human drivers on the road are gonna pay attention to and that and that robots won't so if you put a green
A fake green light it's a person's gonna think it's okay to go or a fake stoplight
The robots not going to see it you can cause an accident
Vegetation is an interesting one because vegetation
Changes so you might often use say a colorized light art and a transmission classifier to determine if a piece of vegetation is
crossable or not right but
But things that overhang and change and grow, like vegetation,
if they're not on your map and it hasn't been updated,
they can cause problems.
So dependence on the map may exacerbate the brittleness
of your sensor discrimination algorithms.
Here's an interesting little video that shows what you can do
if you have a perfect map.
If you have a perfect map,
you don't have to have any kinds of traffic lights.
You can just drive straight through an intersection.
If you have perfect knowledge of the world,
then you can make these split-second decisions
that are always right.
But you don't always have perfect knowledge.
The map has to be constantly updated.
So a local map is vulnerable to unexpected real-world features,
and a remote map is vulnerable to denial of the update process,
so jamming, spoofing, man-in-the-middle attacks, and so on.
So the map is an interesting vulnerability.
And an attacker is going to try and exploit
the logic structure that your vehicle has.
And so the goal is to maximize the uncertainty of the system.
For example, a blind right-hand turn is a very brittle traffic maneuver,
and this is an area where you can force the vehicle
to require manual assistance, confuse and annoy the occupants,
inconvenience the other road users.
If you can prevent the robot from successfully navigating
a fragile maneuver.
And, of course, the best thing to do, if you're an attacker,
you have access to the map as well.
You can find exactly where these fragile decision points are
and cause bad behavior and maybe even cause a bad accident like this.
Trapping and redirecting are a general class of attacks
that attack at the collision avoidance and navigation layers,
forcing the robot to postpone its high-level tasks.
So you can use moving obstacles.
You could perhaps even use robot swarms, obstacle swarms.
You can use artificial traffic control features
to force the robot to go somewhere that's under the attacker's control.
And you can make things,
that are subtle,
that are ways that a human driver would just feel free to ignore them,
but the robot has very, very fixed rules,
and it can't ignore the attacking features
that you're putting in its way.
The other big attack area is clobbering,
so forcing the robot to crash into something,
subverting the collision avoidance layer.
So the goal here is to incapacitate the vehicle,
to strip off sensors or to damage them physically.
You can do this with subtle map deviations
if the robot is very dependent on the map.
You can imitate vegetation that it thinks is crossable, but is not.
You can simulate high-impact obstacles while the vehicle is at speed
that'll cause it to make defensive maneuvers that will cause an accident.
You could disguise entrances like overhangs or gateways
that are within the GPS noise level,
so that the vehicle might scrape its side,
removing those encoders for wheel odometry or so on.
And then,
placing dynamic obstacles underneath large signs to fool the radar.
So all of that said,
there's some debate about whether we'll see level four
and five autonomous vehicles in widespread use on the road anytime soon.
This is a funny quote from 2014,
talking about how 99% of the country is areas
where the autonomous vehicles cannot drive.
So another area that I just wanted to cover very briefly at the end here
is the recent announcement
of the rollout of standards for vehicle-to-vehicle
and vehicle-to-infrastructure communications.
This is from the U.S. Department of Transportation.
These are little radio messages that cars transmit between each other
and between infrastructure that warn them of certain safety hazard things.
So the early V2V components are going to be rear-end collision warning,
lane change warning, and intersection warning scenarios.
Right now, just warnings.
No active control of the vehicle is proposed.
So it uses these radio messages both on board and to roadside communicators.
The protocol is called DSRC, dedicated short-range communications.
These are omnidirectional 200 to 500 byte packets,
and the protocol is called the BSM, basic safety message.
These packets are not encrypted, but they are authenticated via certificates.
So standard PKI issues apply.
Now, these are what the packets look like.
Part one is the core that's transmitted with every packet.
Part two is appended whenever it's changed,
and there's a lot of vehicle-specific information in part two.
GPS is transmitted unencrypted.
So if you can receive these messages from a vehicle,
you can tell where it thinks that it is.
So if you're sending out your GPS spoofing, you can actually get feedback to make sure
that the vehicle actually is being fooled.
Here's a breakdown of the security model.
I want you to see this.
I want you to pay special attention to the stuff that's initial deployment versus full deployment.
So those dashed lines are the initial deployment.
Most of this is slated for later on.
The initial deployment is very small.
A lot of functions are split for security and privacy reasons.
Privacy has been a big component of the development here.
The certificates come pre-installed.
So your car will come with thousands of certificates.
And they're valid for five minutes each.
There's a location obscurer.
So privacy, again, very important here.
The bottom line on V2V, the rollout is very careful.
It was 11 years in development.
They're estimating 37 years to the full fleet.
OK, so that's a long time.
Compare that with all the hype about autonomous vehicles and like, oh,
we're going to have robot taxis by 2022, maybe in some limited cases.
But just for this short range,
if you look at the communications protocol, 37 years to full fleet rollout.
And tracking and privacy are more immediate concerns than other malicious attacks.
The standard PKI concerns apply.
And remember, certificates are going to be very, very easy to get.
You're going to have thousands of them in every vehicle.
But five minutes is a long time on the road.
And there's no direct control even proposed.
So actually, the driverless vehicles may be with us before we have V2V direct control,
if it's going to take us 37 years just for warnings.
But hopefully, they won't make the same mistakes that were made
in other traffic and infrastructure sensors.
So here's some stuff from DEF CON 22 from IOactive.
They showed that these traffic sensors that were sending data
into traffic control systems had no encryption or authentication.
So the messages were clear text wireless transmissions that were fully spoofable.
And firmware updates were neither encrypted nor signed.
So hopefully, they won't make these.
They will make particular specific mistakes, but no doubt, they will make others.
And we as hackers have a duty to find out what they are and notify so that things can
be changed and made safe.
So here's just a quick example, very recent research.
This was released by Georgia Tech in March, I think.
They just looked at what are the results if we had widespread driverless vehicles that
were hackable.
And so this is looking at Manhattan Island, New York.
And just looking at what happened in terms of access to the grid network, right?
If we had a lot of vehicles because we need emergency vehicles to get through.
We need ambulances and fire and so on.
We want our roads to have full access.
But hacking just 10% of the vehicles at rush hour, you can see they used a percolation
theory to develop this simulation.
And you can see a large chunk of the island is now completely inaccessible to any vehicle just by being
able to do denial of service attacks on about 10% of the rush hour vehicles, 10 hacked vehicles
per kilometer per lane.
And then doubling that to 20% and we can see that the island is basically unnavigatable.
So that's really, really serious.
So this is why we need to take this problem seriously and why we need to do research here
and make sure that we're putting these IoT systems out into the world in a way that's
going to keep people safe.
So that's it.
In summary, I will just say I want you to remember that driverless vehicles are cool.
We're in favor of driverless vehicles.
We want all the advantages that they offer.
We're not trying to stop this in any way.
Don't do any of these things in the real world.
Only do it in a research context.
Don't hassle the Hoff or rather don't hacksaw the bots.
And hopefully we can bring the world together into the place where we're living the driverless
dream where we can be doing our romance in the backseat of the car while the robot does
the driving.
Thank you very much.
